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We present a new block-based motion 
estimation strategy, which is able to correctly find 
the velocity of background block? in occlusion areas 
(covering or uncovering). For c o v er in g, we perform 
the Tmrtrfr^c for blocks in the fixture picture, of the 
picture pair used for the motion estimation! fetch- 
ins th * corresponding blocks from the past picture. 
In uncovering areas of the picture we match the 
blocks in the past picture with, the blocks fetched 
from the future image. For the rigid motion of 
blocks in non occlusion areas* we fetch the blocks 
bidirectionalry from both the past and future pic- 
ture, projecting to the interpolated picture halfway 
between the past and future pictures. We then re- 
time the obtained vector field so that it corresponds 
to the desired time moment, of the interpolated pic- 
ture. The result of these two operations is a vector 
field that fits the image objects very well, whereas 
non "occlusion robust* methods can not estimate 
the background vectors correctly In the occlusion 
areas, which typically leads to foreground vector 
field regions that are larger than the corresponding 
image objects. In up conversion this leads to an an- 
noying halo around the foreground objects, which 
can manifest itself as a noisy flicker, blur or parts 
of the background attached to the foreground ob- 
jects. Occlusion is a problem that plagues all block 
based motion estimation m et hods, and hence we see 
a utility of the method for applications like video 
compression, 3D matching or image sequence ob- 
ject extraction. 



1 Introduction 

For real-time motion estimation we have developed the 
3D recursive search block matcher [1], which in con- 
trast to the full search block roatcfaer [2] does not have 
Co evaluate all possible motion vectors within certain 
limits, but only a number of smartly chosen candidates. 



We determine which of these candidates describes the 
motion of the block under study the best, and this 
usually yields a high quality vector field. The criterion 
which determines which motion vector among the pre- 
sented candidates is the best one, is the nrinfmisarion 
of the sum of absolute differences or SAD (eo, 1): 

to 

In this equation B(X) is the block of selected pixels 
at position (ox » 0...8 by Sy = 0...8), O the 
candidate vector under scrutiny, n the picture number 
or moment in time and a a c onst ant, 0 < a < 1, de- 
termining the temporal position between the two pic- 
tures, to which the fetched blocks are projected for the 
matrft In [1] we performed this match on the temporal 
position where the vector field was desired, hence the 
position halfway in between the two selected pictures, 
(a = 0.5), which resulted in impossible matches in the 
occlusion regions, because in one of the pictures the 
background blocks are covered. 

2 Description of the new motion 
estimator strategy 

2.1 The tritemporal motion estimator 

We reafiaed that in the case of e.g. covering, all pixel 
bloclfs that are present in and around the occlusion 
area in the future picture can also be found in the 
past picture. However in the past picture there are 
extra blocks, which no longer exist hi the future pic- 
ture, because they become covered. A natural position 
to put the reference position a for block matching is 
at the temporal position of the future picture (asl), 
since then for all blocks in and around die occlusion 
area a correct motion vector can in principle be found. 
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Pig. la shows more clearly what happens for all pos- 
sible reference positions 0 $ a < L Suppose that the 
upper Hne a corresponds to the position of the edge of 
the foreground object travelling through time. Then 
in the future picture we find the correct foreground 
or background velocities on either side of the velocity 
vector edge position Xb (position where the motion 
vector displays a significant spatial change)* But the 
more we move the reference position <x towards the 
past picture, the larger the ambiguity region becomes, 
in which it is impossible to find the correct vector by 
matching. Since afl candidate vectors will have a large 
SAD value, minimisation randomly selects a certain 
candidate, which means that the resulting vector fields 
are almost impossible to correct* 
Similarly from fig. lb we learn that for uncovering 
regions an unambiguous motion estimation results if 
we put the reference position a equal to 0, in other 
words we project blocks from the fixture to the past. 
For rigid motion no errors occur if we put the reference 
position in the middle (a =3 0*5). The decision whether 
an area is a covering or uncovering area is done based 
upon the sign of the difference of the motion vectors 1 
x~ooxnponents taken from either side of the vector edge 
Xb hi the previously calculated vector field, according 
to the ocdusion detector we described in [3]. 

The vector field obtained by locally rhanging the ref- 
erence position a for block matching according to the 
output of the ocdusion detector, is a motion vector 
field **** is not valid for any temporal position be- 
tween the past and current pictures, since it is in fact 
obtained by "guiag together* sub vector fields which 
are valid at differen t temporal positions {bitemporal)* 
Therefore the vector field has to be retimed to the de- 
aired up conversion position (typically a = 0.5). 



2.2 The retimer 

In the motion vector field originating from the 
tritemporal estimator, all vectors in the interpolated 
picture between Xb and the true edge position of the 
foreground (the point on Hne a at the time n + a) 
are incorrect. In the covering example we described 
with the aid of fig. la, that they will be foreground 
vectors but should be background vectors. The 
retimer performs two actions. First it determines the 
true edge position of the foreground by calculating 
the intersection of the line projecting Xb with the 
foreground velocity, and the interpolated picture 
plane* Secondly it determines whether the foreground 
or background should be assigned to the blocks in the 
incorrect region, depending on: 
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1. whether we have covering or uncovering 

2. the sign of the foreground velocity 

3. on which side the foreground 2s 

It is obvious that for both actions we need to know 
which velocity is the foreground velocity and which 
one the background velocity. For this we developed 3 
foreground/background determination strategies. The 
one we prefer to use projects Xb with both veloci- 
ties (on either side of Xb) to the previous vector field 
(at time n for the covering example), and determines 
around both positions whether there is a vector edge. 
The velocity that projects to the edge is the foreground 
velocity. 

As a practical illustration fig. 2 shows how well the 
retimed vector field fits to the image object. Notice 
that the part of the background that is selected (by 
using the motion vector image as a selection mask) in 
the new method is much smaller, especially behind the 
head, which in our video rate conversion application 
results in a drastically reduced halo around the head. 
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Figure 2: Left: The Renata object extracted hum the image with the calculated vector field for dftfftfmT aon occlusion 
robtast processing (or = 0.3 always). Right: Renata extracted with the vector field obtained from the new motion estimator. 
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The invention can be applied in various areas. Examples are given below. 

In picture rate conversion, where an improved vector field according to the invention 
results in a more pleasing, artefact free video stream. Typical devices for this 
application are TV and PC 

In 3D disparity analysis, where images are generated by a rolling camera or from a 
rolling scene. 

In motion based video compression, where an improved vector field according to the 
invention results in higher quality predictions and therefore in higher compression 
ratios or improved picture quality. An example is compression in MPEG. 

In motion based image analysis, where an improved vector field according to the 
invention results in a more faithful extraction of objects and hence in easier post- 
processing. Examples are: security camera video analysis, video special effects, and 
traffic analysis. 

Picture enhancement in television techniques, e.g. avoidance of blurring of 
background 

In scientific application, where an unproved vector field according to the invention 
results in a better data analysis. Examples are satellite photos of clouds and 
oceanography. 
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A new block matching motion estima- 
tor proposed, enabling a superior performance 
for applications in picture rata conversion* Partic- 
ularly, the invention describes a modification to the 
match error calculation, which enables the estima- 
tioa of a more precise motion vector in areas of 
covering and uncovering* tending to reduced halo 
artifacts in interpolated images. The essence of the 
invention consists of shifting the match error calcu- 
lation from the current picture pair to the previous 
picture pair or to the ne$t picture pair, for image 
parts that are being covered, or uncovered, respec- 
tively. 

Keywords: motion estimation, block matching, 
video format conversion, halo reduction* 



1 Prior art 

In pj a method was disclosed for motion compensated 
picture interpolation that reduces the negative effect 
of covering and uncovering, near discontinuities in the 
velocity field, on the quality of interpolated images. 
In the described case, which applies an order statisti- 
cal filter in the upconveraion to replace the common 
MC-averaging, interpolated pictures result from pixels 
taken from both adjacent pictures. The current inven- 
tion is an elaboration of this original idea, and basi- 
cally Tnpjffipg the interpolation strategy depending on 
a segmentation of the image in various areas. 

In [2] a segmentation for the same purpose was de- 
scribed. This segmentation is based on a motion de- 
tector, and can only realtee reliable results if covering 
and uncovering occur of stationary backgrounds. The 
current invention claims to be valid even if both fore- 
ground and background are moving. 

In [3] a method was disclosed that allows a reduction of 
halo defects in architectures that enable access to one 



field only, or in systems particularly designed to have 
access to one field only in order to obtain the increased 
resolution of an interpolation according to [4]. 

In [5] a method was disclosed that allows a reduction 
of halo defects in architectures that enable access to 
two fields, where the method relies on the use of two 
motion vector estimators, a so called forward and a 
backward estimator. 

In [6] a method was disclosed that allows identification 
of areas in which covering or uncovering takes place, 
analysing the vector field around ^ i ffion*fafr ft fe3 A 

In the current invention disclosure, we shall use the 
method described in [6] to locally modify the match 
error calculation of a block matching motion estfrnfr. 
tor, in order to arrive at a more precise vector field that 
allows us to interpolate images at temporal positions 
in between input fields, where the interpolated images 
exhibit a reduced halo artifact compared to image re- 
sulting from a non-modified block matcher. 

2 Method according to the 
present invention 

The basic observation from which the current inven- 
tion results is that an estimator estimating motion be* 
tween two successive pictures from a video sequence, 
cannot perform well in areas where covering or un- 
covering occurs, as it is typical for these areas that 
the information only occurs in either of the two im- 
ages. Block matchers, as a consequence wul always 
find large match errors even for the correct vector. It 
is recognized, however, that in the case of covering all 
information in the current picture is present in the pre- 
vioua picture pair, while in the event of uncovering the 
fiezi picture (s), while in an area of uncovering, the cur- 
rent picture rfvwt ainfl all mfromation of the previous 
one (locally around the uncovering area). Ergo, fey 
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modifying the match error calculation, controlled by a 
covering/uncovering detector, e,g. the one di sc los ed in 
[6], from matching the current picture with the motion 
compensated previous picture (covering) to matching 
the previous picture with the motion compensated cur- 
rent picture (uncovering), any ambiguity in the estuna- 
tor can be prevented. This is expected to yield more 
accurate and consistent vector fields and therefor re- 
duced halo. This modification shall be elaborated in 
subsection 2.2. 

As a side effect of this dynamically changing match 
calculation, the resulting unmabiguous vector field is 
no longer valid for one moment in time, let alone the 
moment where the upconversion takes place, but this 
^validity moment' changes depending on the area under 
consideration being a covering, uncovering or simple 
rigidly moving area. This side effect can be eliminated 
with a second ingredient of this disclosure, described 
in subsection 2 .3. 

In the next subsection, we shall elucidate the improved 
motion vector calculation using the full-search block- 
matching motion estimator to calculate the motion 
vectors. This algorithm is not a very cost-effective 
approach to the motion estimation problem, but the 
more effective ones, e.g. [7, 8] are based on the same 
principle. 

2.1 The full search block matcher 

Tn block-matching motion estimation algorithms, a dis- 
placement vector is assigned to the centre X = ^jf*) 

of a block of pixels B(£) in the current picture n 
}jy o»fti*»fev»g for a similar block within a search area 
SA{X) t also centred at X % but in the previous pic- 
ture n - 1. The similar _bloek has a centre, which is 
shifted with respect to X over the displacement vec- 
tor D(X,n),T6 find D(X 9 n) 9 a number of can d ida te 
vectors C are evaluated by applying an error measure 
(<7,J?,n) to quantify block similarity- Figure 1 illus- 
trate? the procedure. More formally, CS™** is defined 
as the set of candidates C, describing all possible (usu- 
ally integer) displacements with respect to X within 
the search area SA{X) in the previous image: 

CSP^^ffi-NZCxZN* -m<c v <>m) 

where N and M are constants limiting SA(X), A block 
B(X) centred as £ and of size X by Y, consisting of 

pixel positions S — ( * ) in the present picture n, is 
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Picture number 



Figure 1: Full-search block-matching. 

defined: 

B{t) m {S\X 9 -X/2^rSJf a + X/2A m 
X»-Y/2<y<X 9 + y/2> w 

The displacement vector D(X y n) resulting from the 
block-matching process, is a candidate vector d 
which yields the minimum value of an error function 

D(X,n) € 

If, which is the common case, the vector D{z>n) with 
the smallest matching error is assigned to an pixel po- 
sitions x in the block B(£)z 

V* € B(je> : S(£,n) = 3{X> n) (4) 

rather than to the centre pixel only, a large reduction 
of the number of computations is achieved* 

As an implication, consecutive blocks B(X) are not 
overlapping. The error value fox a given candidate vec- 
tor C is a function of the luminance values of the pixels 
in the current block and those of the shifted block from 
a previous picture, summed over the block B(X). A 
common choice, which we too shall use, is the sum of 
the absolute differences: 

£(C,.Y,n) ss 

ZmJh - oC 9 n - 1) - (1 - ft )0,n)| 

(5) 

where a is a constant, 0 £ a £ 1, determining the 
temporal position between the two pictures, where the 
vector field has to be valid. 
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numbtr (b) 

Figure 2; In case of covering, (a), unambiguous motion estimation can result only by **i*ni»Hn e the match errors by 
shiftin g a previous picture towards the current, as only the previous picture contains all W^rmfrtfrrn present in the entreat. 
In case of uncovering, (b), unambiguous motion estimation can revolt only by calculating the match errors by shifting a 
current picture towards the previous, as only the current picture contains aU information present in the previous. 



2.2 Modification of the FS block 
matcher to prevent ambiguities 



The improvement proposed, results from the observa- 
tion that in areas where covering occurs, correct mo- 
tion vectors can only be estimated with an estimator 
using asL Similarly, for uncovering only correct 
vectors are found with an estimator using a = 0. 

Vector estimators calculating vectors for intermediate 
temporal instances have difficulties with both cover- 
ing and uncovering, be it that their worst case area of 
ambiguity is smaller than for the extreme estimators. 
(The worst case ambiguity is least for the estimator 
applying a = 0.5). The Figures 2a and 2b illustrate 
the effect of a on the size of the ambiguous area. 



The first step in improving the block matcher for cov- 
ering and uncovering situations, regardless the required 
value of a for the interpolated picture, results by change 
ing equation 5 to: 

Skkii r<*- d > n - 1 > - *»»>| c ' 

in case an area of covering, as indicated by the cov- 
ering/uncovering detector, e.g. the one described in 
[6]and to 

|W.n - 1) - F(aT+ (?.n)| 

in the event of uncovering. The covering/uncovering 
detector may use a previous vector field, or a previa 
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ous calculation (iteration) of the current vector field, 
or use methods based on evaluation of match errors 
calculated between neighbouring pictures. 

As a consequence of the proposed modification, there 
are no ambiguities for the motion estimator, although 
we have sacrificed the correct temporal instance where 
the vector field is valid. This shall be detailed in the 
following sub-section* 

2.3 Calculating the required vector 
field for up conversion from the un- 
ambiguous vector field 

In a second step of the algorithm, the retimer, the time 
error is corrected. To this end, the vector field is 'pro- 
jected' to the desired temporal instance, Le. projected 
back in lime for covering: 

Zr(£,n-l + a)=:P(x-h(l^a)5(^n),n) (8) 

and forward for uncovering: 

D*(x 9 n-l + a) = D(g~a3{S t n-l) t n-l) (9) 

This projection reintroduces ambiguous areas, Le. ar- 
eas to which no vector is assigned. The origin of this 
ambiguity is that with one estimate it is unknown 
whether the discontinuity has moved along the Kne 'a', 
or along the line V in Figure 2, hence it is not trivial 
that the vector taken from e.g. position x above the 
discontinuity is the background vector which should 
be put in thi$ position in the upconversion vector field. 
With help of the previous vector field however, it is pos- 
sible to judge with which of the two possible vectors 
the edge moves, and hence which is the background 
and which the foreground velocity. 

The retdmer first determines the starting position *a 
of the ambiguity area in the upconversion vector field 
at time n — 1 + a by projecting the foreground velocity 
from the vector edge at position x& in the unambigu- 
ous vector field. A small refinement results if the posi- 
tion a£t of the edge in the upconversion vector field is 
not a result of shifting the estimates from one vector 
field, but is calculated as the weighted (with or) aver- 
age of the positions in the current and previous vector 
field. The retdmer then fills the space between x\ and 
xb with either the foreground or background velocity, 
depending on the side of the foreground region (left or 
right of xb) and the sign of the foreground velocity. For 
this strategy to work, a robust foreground/background 
determination strategy is needed. We describe three 
strategies* 



In a first strategy, the average vector fore* 
ground/background determination, we make use of the 
fact that any vector v fetch = J»FC+(l-fc)vjrc, where 
k is smaller than 1 and vj?G and vSg are the velocities 
of the foreground and background objects at position 
£ , fetches a background velocity from the previous vec* 
tor field in the case of covering and a foreground veloc- 
ity in the case of uncovering. The safest vector to use 
is the average vector vZv =* Q'Svpq+Q&vbg* More for- 
mally, we calculate the two possible positions S a and 
3* of the edge in the previous image pair, e.g. for cov- 
ering and a vertical edge at position z& in vector field 

7X1 

< . >?< CIO) 

Xi = XE + D(£+ ^ Q J ,n) 

and fetch the vector present at an intermediate position 
in the previous vector field (covering) in the ambiguouB 
ana: _ _ 

&«&n)^D(Z£±^,n-l) (11) 

If we need to fill in the foreground vector in the am- 
biguous area of the interpolation vector field, we choose 
between S a and z* the one which is most different from 

A variant of this first strategy fetches the background 
vector from the future for uncovering: 

A,(*.n) = £(3s±*,») (12) 

with: 

- ft - (J),n - 1)5* = ft - D{S + (j),n - 

(13) 

A second strategy, the twosided self speed fore- 
ground/background determinatioTh uses the fact that 
for uncovering, positions projected to the past with the 
background velocity have a higher probability of cross- 
ing towards the foreground region than when they are 
projected with v* v . This is interesting when small rel- 
ative velocities vfq — v§g or inaccurately estimated 
vector fields occur. Because we do not know a priori 
which velocity is the background velocity, we project 
the two positions on either side of the edge with its own 
velocity VaTi/ (see figure 3)- As we can see for the ideal 
case, the lower velocity changes from the background 
velocity at n to the foreground velocity at n — 1. The 
probability that a block in foreground in n projects 
to foreground in n-1 is so high that for practical rea- 
sons we can consider it to be 1. AH the other probable 
decisions are shown in table 2.3* 
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Upper block is in reality; 


fc^n'l u J-ll-! : ! : **N 1Uf 'l 


Decision for upper block 


PG 


PG or BG 


Decision for lower block 


PG or BO 


PG 







T^hle 1: Probable deciaion* for the Velocities of the block 
just above and just below the velocity discontinuity far tbe 
second foreground /background determination strategy in 
case of nncuvcring. 




figure 3: Twosdded self 8peed feeground/badcgronnd de- 
termination* 



In case the two projections yield the same (foreground) 
vector, we have a certain determination If this vector 
equals the vector of the upper block, this vector is the 
foreground velocity vector and vice versa, In case the 
vectors are different, the method was unsuccesfcl, and 
yields an uncertain determination. A similar projec- 
tion towards the future can be applied for the case of 
covering. 

A third strategy, the edge projection fore* 
ground/background determination^ checks for e.g. 
covering whether the edge between vpd end v£a in 
the previous image n is present at position a or b 
(see Figure 2). IF the edge is detected at position 

a, vfc - D{£- (o)> n + 1) and vice versa. Care 

should be taken that the velocities in n are the same 
velocities as in n + 1, since other velocity edges can 
occur in the vicinity of the projection. Obviously 
A gftin the principle can be applied by substituting 
uncovering for covering and the future for the past. 

The most natural strategy, if one only wants to use 
vector image from the past, is to use the second fore- 
ground /background determination strategy for uncov- 
ering and the third one for covering. Alternatively 
one can use the first strategy for both. It should be 
noted that the strategies can be whanred by check- 
ing the match errors. In case a crossing to the fore* 



ground region occurred, the match errors of the vector 
in that block should be low. In case we project to a 
background region that was erronously allotted a fore- 
ground vector in the previous image, the errors should 
be higher. 



3 Claims I 

Claim 1: A method, and apparatus realizing this 
method} for detecting motion between previous 
and next images by optimising a criterion function 
for candidate vectors the function further depend- 
ing on data from a previous image shifted over a 
fraction a times the candidate vector and data 
from a next image shifted over oc — 1 times the 
candidate vector, characterised in that the frac- 
tion a may change within the picture period. 



Claim 2: A method, and apparatus realizing this 
method, according to claim 1, in which the afore- 
mentioned criterion function is a match error, e.g. 
a sum over a block of pixels of absolute pixel dif- 
ferences, and the optimisation concerns a minimi- 
sation of this sum. 

Claim 3: A method, and apparatus realizing this 
method, according to the previous claims, in 
which the fraction a, is controlled by a cover- 
ing/uncovering detector. 

Claim 4: A method, and apparatus realizing this 
method, according to any of the previous claims, 
in which the fraction a i$ set to 1 in case of cov- 
ering, and sec to 0 in case of uncovering. 

Claim 5: A method, and apparatus realizing this 
method, according to the previous claims, in 
which the aforementioned covering/uncovering de- 
tector decides on data in a previous vector field to 
adapt the fraction or in the current estimation- 



4 Claims II 

In the next set of claims we describe how a nniftfld 
vector field can be calculated from (the) available 
vector fields. It is unclear to us whether this can 
be part of the same patent application, or should 
be filed separately. 

Claim 1: A method, and apparatus realizing this 
method, to calculate a vector field, valid at a 
temporal distance <x times the field period from 
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the previous picture, from neighbouring vector 
fields by projecting motion vectors, characterized 
in that the projection direction changes within one 
output vector field. 

Claim 2: A method, and apparatus realizing this 
method, according to the previous claim, in which 
the projection direction is controlled by a cover* 
ing/unoovexing detector. 

Claim St A method* and apparatus realising this 
method, according to the previous claim, in which 
the projection direction points from a next vector 
field to the required vector field in case of coveting 
and from the previous vector field to the required 
in case of uncovering. 

Claim 4: A method, and apparatus realizing this 
method, according to the previous claims, in 
which at the position xi of a discontinuity in the 
vector field 

• a first position in the previous (covering) 
or next (uncovering) vector field is calculated 
by shifting S\ over the first vector at one side 
of the discontinuity 

• a second position 5* in the previous (cover- 
ing) or next (uncovering) vector field is cal- 
culated by shifting S\ over the second vector 
at the other side of the discontinuity 

• and a third intermediate position, e.g. (x a + 
Sb)/2 is calculated 

• while finally, the vector fetched with i£« at 
the third position in the previous (covering) 
or next (uncovering) vector field is filled in 
those regions of the projected vector field 
in the environment of the disconinuity, to 
which no vector ip projected, in case the 
background vector vfo should be filled in. 

The vector chosen between 3(S — ^» n ) 

and 3(x + Q^,n) which is most different 

from tfeu is filled in, in case a foreground vec- 
tor vfo should be filled in. 

Claim 5: A method, and apparatus realizing this 
method, according to the previous claims, in 
which a background velocity is identified as 
a velocity which crosses the velocity discon- 
tinuity and projects to a foreground velocity 
in the previous picture, whereas a foreground 
velocity projects to itself . 
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Claim 6: A method, and apparatus realizing this 
method, according to the previous ri^imc 
which near discontinuities of the vector field 
it is tested whether the mentioned disconti- 
nuity has moved over the first vector on one 
side of the edge, or over the second vector 
on the other aide of the edge. In case the 
edge moves with the first (second) vector, the 
second (first) vector is filled in those regions 
of the projected vector field in the environ- 
ment of the discontinuity, to which no vec- 
tor is projected, in case a background vector 
vbg should be filled in, and the other vector 
is filled mi in case a foreground vector vpg 
should be filled in . 

Claim 7: A method, and apparatus realizing this 
method, according to the previous claims, in 
which the crossing from a background region 
to a foreground region in the previous image 
is verified by the match error of the vector in 
that block. 
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